There are conventional techniques for detecting a predetermined image object from an image (captured image) captured by an image-capturing section such as a camera. For example, Non-Patent Document 1 (Hirokazu Kato, Mark Billinghurst, Koichi Asano, Keihachiro Tachibana, “An Augmented Reality System and its Calibration based on Marker Tracking”, Journal of Virtual Reality Society of Japan, Vol. 4, No. 4, 1999) describes performing an image recognition process on a rectangular marker included in an image captured by a camera in an augmented reality technique. In Non-Patent Document 1, connected regions are extracted by binarizing the captured image using a fixed threshold value, and the outline of a connected region is extracted after excluding those of the extracted connected regions that are different from the predetermined image object (being too larger or too small, for example). Then, the process obtains four line segments from the outline and calculates the vertices of the rectangular marker so as to detect the position of the marker.
With the recognition method described in Non-Patent Document 1, the process detects the position (four vertices) of the image object from the outline of the image object (marker), and therefore the position of the image object may not be detected accurately when the captured image is blurred due to, for example, a camera shake, or the like. That is, if the captured image is blurred, it is possible that the outline of the image object has not been obtained accurately, and therefore it is not possible to obtain the accurate position by calculating the position of the image object from the outline (see FIG. 7).
It is therefore an object of the present invention to provide a storage medium having stored therein an image recognition program capable of accurately detecting the position of the image object even if the captured image is blurred, an image recognition apparatus, an image recognition system, and an image recognition method.
The present invention employs configurations (1) to (16) below to solve the problems mentioned above.
(1)
The present invention is directed to a computer-readable storage medium having stored therein an image recognition program to be executed by a computer of an information processing apparatus for detecting a predetermined image object including a plurality of graphic patterns from a captured image captured by an image-capturing section. The image recognition program instructs the computer to function as: an image obtaining section, a graphic pattern detection section, and an object detection section. The image obtaining section obtains the captured image captured by the image-capturing section. The graphic pattern detection section detects the plurality of graphic patterns from the captured image. The object detection section detects a position or direction of the predetermined image object by calculating center positions of the plurality of graphic patterns detected by the graphic pattern detection section.
The “image-capturing section” may be built in the information processing apparatus, or may be a separate unit from the information processing apparatus. Where the image-capturing section is built in the information processing section, the image-capturing section may be fixedly attached to, or detachable from, the information processing apparatus. The image-capturing section may operate in cooperation with, or independently of, the operation of the information processing apparatus.
While the “predetermined image object” is typically a marker, it may be any object as long as it includes a plurality of graphic patterns. While the “graphic pattern” is a plurality of circular graphic patterns in the embodiment to be described later (FIG. 6), the shape, the number, the size and the color thereof are not limited thereto. For example, the “graphic patterns” may be circular, elliptic or polygonal.
While the “graphic patterns” are inner graphic patterns drawn inside the first graphic pattern in the embodiment to be described later, they do not have to be drawn inside a graphic pattern. That is, the “graphic pattern detection section” may detect a plurality of graphic patterns from within the area of the first graphic pattern (after detecting the area of the first graphic pattern) as in the embodiment to be described later or the configuration (2) below, or may detect a plurality of graphic patterns from the entire area of the captured image (without detecting the area of the first graphic pattern).
The “information processing apparatus” is a concept encompassing any computers that process information by executing computer programs, as well as the game apparatus described in the embodiment to be described later. The “information processing apparatus” may or may not be of a portable type.
While the “image recognition program” is, for example, the game program described in the embodiment to be described later, it is a concept encompassing application programs to be executed by personal computers or portable terminals.
The “image obtaining section” may obtain a captured image from an image-capturing section provided in the information processing apparatus via an internal bus, or the like, may obtain a captured image from an image-capturing section provided outside the information processing apparatus via telecommunications, or may obtain a captured image from a storage medium storing therein a captured image captured by an image-capturing section when the storage medium is attached to the information processing apparatus.
The “graphic pattern detection section” may be any section as long as it detects the plurality of graphic patterns, and the specific method for detecting the plurality of graphic patterns may be any method. The “graphic pattern detection section” may detect the area of the graphic pattern as in the embodiment to be described later or the configuration (3) below, for example, may extract the outline of the graphic pattern as in the configuration (4) below, or may detect a graphic pattern by using a pattern matching method.
The “center position of (each) graphic pattern” does not have to be the position of the center of the graphic pattern in the strict sense, but may be a position in the vicinity of the center of the graphic pattern which can be calculated by any of various arithmetic operations. For example, the “center position of a graphic pattern” may be calculated as an average value among pixels representing the outline of the graphic pattern, or where pixels of the captured image are weighted in some manner (with luminance values, etc.), it may be calculated as the center of gravity among pixels within the area of the graphic pattern.
To “detect a position of a predetermined image object by calculating center positions” only requires that the position of the predetermined image object can be identified based on the center positions. Therefore, for example, the “object detection section” may calculate a straight line (approximate straight line) from center positions and identify the position of the predetermined image object based on the approximate straight line as in the embodiment to be described later, the configuration (6) below or the configuration (16) below, or may use the center positions directly as the position of the predetermined image object as described in (Variation Regarding Calculation Of Marker Position) below. Since the position or direction of the predetermined image object within the captured image can be identified from the center positions of the plurality of graphic patterns, it is only required that the object detection section is capable of detecting at least one of the position and the direction.
With the configuration (1) above, the position of the predetermined image object is detected based on the center positions of the graphic patterns, not on the outline of the first graphic pattern. Since it is difficult to accurately detect the outline of the predetermined image object if the captured image is blurred, it is not possible to accurately detect the position of the predetermined image object from the outline. In contrast, the center position can be calculated accurately even if the captured image is blurred. Therefore, with the configuration (1) above, it is possible to accurately calculate the position or direction of the predetermined image object even if the captured image is blurred.
(2)
The predetermined image object may include a first graphic pattern with the plurality of graphic patterns drawn therein. Then, the image recognition program instructs the computer to further function as an area detection section for detecting an area of the first graphic pattern from the captured image; and. The graphic pattern detection section detects the plurality of graphic patterns from within the area detected by the area detection section.
The “area detection section” may be any section as long as it detects the first graphic pattern of the predetermined image object, and the specific method for detecting the first graphic pattern may be any method. For example, the “area detection section” may detect the first graphic pattern by extracting the outline of the first graphic pattern, or may detect the first graphic pattern by using a pattern matching method.
With the configuration (2) above, first, the area of the first graphic pattern is detected, and the graphic patterns (inner graphic patterns) drawn inside the first graphic pattern are detected from within the detected area. Then, it is possible to prevent an erroneous detection of detecting an object that is not the predetermined image object as being the inner graphic pattern, and it is possible to reliably detect the inner graphic patterns.
(3)
The graphic pattern detection section may detect areas of the plurality of graphic patterns. Then, the object detection section includes a position calculation section for calculating center positions of the areas detected by the graphic pattern detection section.
With the configuration (3) above, center positions can be easily calculated by detecting the areas of the plurality of graphic patterns and then calculating the center positions of the detected areas.
(4)
The graphic pattern detection section may detect outlines representing the areas of the plurality of graphic patterns. Then, the position calculation section calculates center positions of the areas represented by the outlines detected by the graphic pattern detection section.
With the configuration (4) above, center positions can be easily calculated by detecting the outlines of the areas of the plurality of graphic patterns and then calculating the center positions of the detected outlines.
(5)
The position calculation section may calculate the center position of each of the areas by obtaining positions of pixels representing the outline detected by the graphic pattern detection section and calculating an average value among the obtained positions.
With the configuration (5) above, the center position can be calculated easily and accurately by calculating the average value among the positions of pixels representing the outline.
(6)
The object detection section may include a straight line calculation section for calculating a predetermined straight line based on the center positions of the plurality of graphic patterns. Then, the image recognition program instructs the computer to further function as a positional relationship calculation section for calculating a positional relationship between the predetermined image object and the image-capturing section based on the predetermined straight line calculated by the straight line calculation section.
With the configuration (6) above, the position and direction of the predetermined image object in the captured image can be identified by the straight line calculated based on the center positions. Therefore, the positional relationship can be calculated accurately based on the straight line.
With the configuration (6) above, the virtual image of the embodiment to be described later can be synthesized with the captured image by using the positional relationship. That is, with the configuration (6) above, in an augmented reality technique for displaying a synthesized image obtained by synthesizing together the captured image and the virtual image, it is possible to use a recognition process that is resistant to the blur of the captured image (capable of accurately detecting the predetermined image object even if the captured image is blurred). Note that in an augmented reality technique, it is assumed that it is often the case where the user uses the image-capturing section while moving it around, and it is believed that that is particularly often the case when the augmented reality technique is used in a game as in the embodiment to be described later. Therefore, where the configuration (6) above is used in an augmented reality technique, the recognition process of the configuration (1) above which is resistant to the blur of the image is particularly effective.
(7)
Each of the plurality of graphic patterns may be circular.
With the configuration (7) above, the shape of the graphic pattern in the captured image remains the same irrespective of the direction in which the predetermined image object lies. Therefore, the center position does not vary depending on the direction of the predetermined image object, and it is therefore possible to more accurately detect the predetermined image object.
(8)
The area detection section may detect an area of the first graphic pattern based on pixel values obtained at a first pitch across the captured image. Then, the object detection section calculates center positions of the graphic patterns based on pixel values obtained at a second pitch smaller than the first pitch across the area of the captured image detected by the area detection section.
The “second pitch” may be of any length as long as it is smaller than the “first pitch”. Where a plurality of different processes are performed in the image object detecting process, as in the embodiment to be described later, the object detection section may perform the process of calculating the center position based on pixel values obtained at the second pitch, while the other processes may be performed with any precision.
With the configuration (8) above, after first detecting the first graphic pattern with a low precision by the area detection section, the process of calculating the center positions of the plurality of graphic patterns within the first graphic pattern is performed with a high precision. Thus, the process of detecting the first graphic pattern is performed with a low precision, and the range across which the process of detecting the plurality of graphic patterns is performed is limited to within the first graphic pattern, and it is therefore possible to perform the recognition process in a short period of time. Since the process of calculating the center positions of the detected graphic patterns is performed with a high precision, it is possible to detect the position of the predetermined image object precisely. Therefore, with the configuration (8) above, it is possible to provide a recognition process with which it is possible to shorten the amount of time required for detection while maintaining the object detection precision.
(9)
The image recognition program may instruct the computer to further function as an outline extraction section for extracting an outline of the first graphic pattern in the area of the first graphic pattern detected by the area detection section. Then, the object detection section detects the plurality of graphic patterns from an area within the outline extracted by the outline extraction section.
With the configuration (9) above, the graphic patterns are detected from within the area of the outline extracted by the outline extraction section, and it is therefore possible to appropriately determine the range across which the detection of the graphic patterns is performed. Thus, it is possible to prevent the graphic pattern detection process from being performed wastefully on unnecessary areas, and it is therefore possible to more efficiently perform the recognition process.
(10)
The outline extraction section may extract the outline of the first graphic pattern based on pixel values obtained at a predetermined pitch across the captured image. Then, the object detection section detects the plurality of graphic patterns from the area within the outline based on pixel values obtained at a pitch smaller than the predetermined pitch across the captured image.
With the configuration (10) above, the outline extracting process is performed with a lower precision than the graphic pattern detection process, and it is therefore possible to perform the outline extracting process in a shorter period of time. As a result, it is possible to perform the recognition process in a shorter period of time.
(11)
The image recognition program may instruct the computer to function as a shape determination section for determining whether the outline extracted by the outline extraction section represents a shape of the first graphic pattern. Then, the object detection section detects the plurality of graphic patterns from within an area corresponding to the outline which is determined by the shape determination section to represent a shape of the first graphic pattern.
With the configuration (11) above, it is possible to determine, based on the extracted outline, whether the first graphic pattern is included within the area detected by the area detection section. Then, where the first graphic pattern is not included within the area, the graphic pattern detection process is not performed for that area. Therefore, with the configuration (11) above, areas that do not include the first graphic pattern can be excluded from the graphic pattern detection process, and it is therefore possible to prevent the graphic pattern detection process from being performed wastefully. Thus, it is possible to efficiently perform the recognition process and to perform the recognition process in a shorter period of time. Particularly, where the configurations (10) and (11) above are combined with each other, the graphic pattern detection process which requires a relatively long time (as it is performed with a high precision) can be omitted by virtue of the outline extracting process which is performed relatively quickly (as it is performed with a low precision), and it is therefore possible to perform the recognition process more efficiently.
(12)
The predetermined image object may include a second graphic pattern at a position different from the first graphic pattern. Then, the image recognition program instructs the computer to further function as a direction identification section for detecting the second graphic pattern from the captured image and identifying a direction of the predetermined image object based on a direction in which the second graphic pattern is arranged with respect to the first graphic pattern.
With the configuration (12) above, the direction of the predetermined image object can be identified based on the direction of the second graphic pattern with respect to the first graphic pattern. With the configuration (12) above, since the second graphic pattern is formed at a position of the predetermined image object different from the first graphic pattern, it is possible to have only the inside graphic patterns drawn inside the first graphic pattern. Therefore, when detecting the inside graphic patterns, it is possible to accurately detect the inside graphic patterns without erroneously detecting the second graphic pattern or a part thereof. Therefore, it is possible to further improve the precision of the recognition process.
(13)
The graphic pattern detection section may detect the areas of the plurality of graphic patterns based on pixel values obtained at a predetermined pitch across the captured image. Then, the position calculation section calculates center positions of the areas detected by the graphic pattern detection section based on pixel values obtained at a pitch smaller than the predetermined pitch across the captured image.
With the configuration (13) above, since the process of detecting the areas of the plurality of graphic patterns is performed with a lower precision than the center position calculation process, it is possible to perform the process of detecting the areas of the plurality of graphic patterns in a shorter period of time. As a result, it is possible to perform the recognition process in a shorter period of time.
(14)
The object detection section may further include a number-of-areas determination section for determining whether the number of areas detected by the graphic pattern detection section is a number within a predetermined range. Then, the position calculation section calculates center positions of the areas detected by the graphic pattern detection section for the area of the first graphic pattern where a number of areas within the predetermined range have been detected.
With the configuration (14) above, it is possible to determine whether the number of areas which have been detected by the graphic pattern detection section is an appropriate number. If the number of areas is not appropriate, it is determined that the first graphic pattern is not included in the area detected by the area detection section, and the process of calculating the center positions is not performed. Therefore, with the configuration (14) above, the areas that do not include the first graphic pattern can be excluded from the center position calculation process, and it is therefore possible to prevent the calculation process from being performed wastefully. Thus, it is possible to efficiently perform the recognition process and to perform the recognition process in a shorter period of time. Particularly, where the configurations (13) and (14) above are combined with each other, the process of calculating the center positions which requires a relatively long time (as it is performed with a high precision) can be omitted by virtue of the process of detecting the area of the first graphic pattern which is performed relatively quickly (as it is performed with a low precision), and it is therefore possible to perform the recognition process more efficiently.
(15)
The predetermined image object may include three or more graphic patterns. Then, the object detection section detects a position and direction of the predetermined image object based on center positions of the three or more graphic patterns.
To “detect the direction (of the predetermined image object)” is not limited to uniquely identifying the direction of the predetermined image object. For example, while the process of step S26 calculates the positions of the four vertices from the center positions in the embodiment to be described later, it is not possible, based on the positions of these four vertices, to distinguish between the marker 51 being rotated by 90°, 180° or 270° and being not rotated. The object detection section in the configuration (15) above may be a process that does not uniquely identify the direction of the predetermined image object, as with the process of step S26 described above.
With the configuration (15) above, it is possible to identify the positions (in the captured image) of three points in the predetermined image object by calculating the center positions of the three or more graphic patterns, and it is therefore possible to detect the position and direction of the predetermined image object.
(16)
A predetermined number, three or more, of graphic patterns may be arranged on a predetermined straight line in the predetermined image object. Then, the object detection section calculates the predetermined straight line based at least on a number of center positions among all center positions calculated, the number being two or more and smaller than the predetermined number, so as to detect a position and direction of the predetermined image object based on the calculated predetermined straight line.
With the configuration (16) above, it is possible to detect the position and direction of the predetermined image object based on the position and direction of the predetermined straight line in the captured image. Herein, the predetermined straight line is calculated from the center positions of a number of graphic patterns which is less than a predetermined number, and it is therefore possible to calculate the predetermined straight line even if a predetermined number of graphic patterns are not all detected. Therefore, with the configuration (16) above, there is provided redundancy in the number of graphic patterns, whereby it is possible to detect the position and direction of the predetermined image object even if some of the graphic patterns are not detected. Therefore, it is possible to more reliably detect the predetermined image object, and thus to increase the success rate of the recognition process.
The present invention may also be embodied in the form of an image recognition apparatus including various sections similar to those described above. In such an image recognition apparatus, the various sections may be implemented by a computer executing an image recognition program, or some or all of the various sections may be implemented by a dedicated circuit or circuits. The present invention may also be embodied in the form of an image recognition system including one or more information processing apparatuses each having various sections described above. Then, the one or more information processing apparatuses may communicate with each other directly via a wired or wireless connection, or may communicate with each other via a network. Moreover, the present invention may also be embodied in the form of an image recognition method performed by the various sections described above.
The present invention may also be embodied in the form of a marker used in an augmented reality technique. That is, the marker is used in an image display system, in which a captured image captured by an image-capturing section is subjected to a predetermined recognition process to calculate a position and direction of the marker within the captured image, and an image of a virtual object is produced based on the position and direction of the marker, so as to display an image obtained by synthesizing the image of the virtual object with the captured image. The marker includes a plurality of inner graphic patterns. The center positions of the plurality of inner graphic patterns are calculated (by the image display system). The center positions are used for calculating (by the image display system) information representing the position and direction of the marker. The marker may be a thin plate-like member with a graphic pattern drawn thereon as in the embodiment to be described later, or may be implemented by a graphic pattern displayed on the screen of a display apparatus.
According to the present invention, the process calculates the center positions of a plurality of graphic patterns included in a predetermined image object, and detects the position of the predetermined image object based on the center positions, and it is therefore possible to accurately calculate the position of the predetermined image object even if the captured image is blurred.
These and other objects, features, aspects and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings.